Transition build system of cuda_cccl and cuda_parallel to scikit-build-core #3597

oleksandr-pavlyk · 2025-01-30T00:42:51Z

Description

Change build system of internal Python package cuda_cccl that installs CCCL headers into Python prefix.

The advantage of using scikit-build-core is that installation of headers is controlled by project's top-level "CMakeLists.txt"

A minimal pytest-based test suite is added as well.

Checklist

New or existing tests cover these changes.
The documentation is up to date with these changes.

Transition from using setuptools to using scikit-build-core, so that CMakeLists.txt is used to ensure that include is the same as in the cuda toolkit. Added test folder to test that the package works correctly.

Use suggestion by Bradley to use scikit-build-core regex facility to extract version value from "_version.py"

github-actions · 2025-01-30T03:43:33Z

🟥 CI finished in 4m 36s: Pass: 0%/1 | Total: 4m 36s | Avg: 4m 36s | Max: 4m 36s

🟥 python: Pass: 0%/1 | Total: 4m 36s | Avg: 4m 36s | Max: 4m 36s

🟥 cpu
  🟥 amd64              Pass:   0%/1   | Total:  4m 36s | Avg:  4m 36s | Max:  4m 36s
🟥 ctk
  🟥 12.6               Pass:   0%/1   | Total:  4m 36s | Avg:  4m 36s | Max:  4m 36s
🟥 cudacxx
  🟥 nvcc12.6           Pass:   0%/1   | Total:  4m 36s | Avg:  4m 36s | Max:  4m 36s
🟥 cudacxx_family
  🟥 nvcc               Pass:   0%/1   | Total:  4m 36s | Avg:  4m 36s | Max:  4m 36s
🟥 cxx
  🟥 GCC13              Pass:   0%/1   | Total:  4m 36s | Avg:  4m 36s | Max:  4m 36s
🟥 cxx_family
  🟥 GCC                Pass:   0%/1   | Total:  4m 36s | Avg:  4m 36s | Max:  4m 36s
🟥 gpu
  🟥 v100               Pass:   0%/1   | Total:  4m 36s | Avg:  4m 36s | Max:  4m 36s
🟥 jobs
  🟥 Test               Pass:   0%/1   | Total:  4m 36s | Avg:  4m 36s | Max:  4m 36s

👃 Inspect Changes

Modifications in project?

	Project
	CCCL Infrastructure
	libcu++
	CUB
	Thrust
	CUDA Experimental
+/-	python
	CCCL C Parallel Library
	Catch2Helper

Modifications in project or dependencies?

	Project
	CCCL Infrastructure
	libcu++
	CUB
	Thrust
	CUDA Experimental
+/-	python
	CCCL C Parallel Library
	Catch2Helper

🏃‍ Runner counts (total jobs: 1)

#	Runner
1	`linux-amd64-gpu-v100-latest-1`

leofang · 2025-01-30T15:46:14Z

cc @bdice @vyasr for vis

python/cuda_cccl/test/test_cuda_cccl.py

github-actions · 2025-01-30T17:14:20Z

🟥 CI finished in 6m 03s: Pass: 0%/1 | Total: 6m 03s | Avg: 6m 03s | Max: 6m 03s

🟥 python: Pass: 0%/1 | Total: 6m 03s | Avg: 6m 03s | Max: 6m 03s

🟥 cpu
  🟥 amd64              Pass:   0%/1   | Total:  6m 03s | Avg:  6m 03s | Max:  6m 03s
🟥 ctk
  🟥 12.6               Pass:   0%/1   | Total:  6m 03s | Avg:  6m 03s | Max:  6m 03s
🟥 cudacxx
  🟥 nvcc12.6           Pass:   0%/1   | Total:  6m 03s | Avg:  6m 03s | Max:  6m 03s
🟥 cudacxx_family
  🟥 nvcc               Pass:   0%/1   | Total:  6m 03s | Avg:  6m 03s | Max:  6m 03s
🟥 cxx
  🟥 GCC13              Pass:   0%/1   | Total:  6m 03s | Avg:  6m 03s | Max:  6m 03s
🟥 cxx_family
  🟥 GCC                Pass:   0%/1   | Total:  6m 03s | Avg:  6m 03s | Max:  6m 03s
🟥 gpu
  🟥 v100               Pass:   0%/1   | Total:  6m 03s | Avg:  6m 03s | Max:  6m 03s
🟥 jobs
  🟥 Test               Pass:   0%/1   | Total:  6m 03s | Avg:  6m 03s | Max:  6m 03s

👃 Inspect Changes

Modifications in project?

	Project
	CCCL Infrastructure
	libcu++
	CUB
	Thrust
	CUDA Experimental
+/-	python
	CCCL C Parallel Library
	Catch2Helper

Modifications in project or dependencies?

	Project
	CCCL Infrastructure
	libcu++
	CUB
	Thrust
	CUDA Experimental
+/-	python
	CCCL C Parallel Library
	Catch2Helper

🏃‍ Runner counts (total jobs: 1)

#	Runner
1	`linux-amd64-gpu-v100-latest-1`

github-actions · 2025-01-30T19:25:42Z

🟩 CI finished in 47m 48s: Pass: 100%/1 | Total: 47m 48s | Avg: 47m 48s | Max: 47m 48s

🟩 python: Pass: 100%/1 | Total: 47m 48s | Avg: 47m 48s | Max: 47m 48s

🟩 cpu
  🟩 amd64              Pass: 100%/1   | Total: 47m 48s | Avg: 47m 48s | Max: 47m 48s
🟩 ctk
  🟩 12.6               Pass: 100%/1   | Total: 47m 48s | Avg: 47m 48s | Max: 47m 48s
🟩 cudacxx
  🟩 nvcc12.6           Pass: 100%/1   | Total: 47m 48s | Avg: 47m 48s | Max: 47m 48s
🟩 cudacxx_family
  🟩 nvcc               Pass: 100%/1   | Total: 47m 48s | Avg: 47m 48s | Max: 47m 48s
🟩 cxx
  🟩 GCC13              Pass: 100%/1   | Total: 47m 48s | Avg: 47m 48s | Max: 47m 48s
🟩 cxx_family
  🟩 GCC                Pass: 100%/1   | Total: 47m 48s | Avg: 47m 48s | Max: 47m 48s
🟩 gpu
  🟩 v100               Pass: 100%/1   | Total: 47m 48s | Avg: 47m 48s | Max: 47m 48s
🟩 jobs
  🟩 Test               Pass: 100%/1   | Total: 47m 48s | Avg: 47m 48s | Max: 47m 48s

👃 Inspect Changes

Modifications in project?

	Project
	CCCL Infrastructure
	libcu++
	CUB
	Thrust
	CUDA Experimental
+/-	python
	CCCL C Parallel Library
	Catch2Helper

Modifications in project or dependencies?

	Project
	CCCL Infrastructure
	libcu++
	CUB
	Thrust
	CUDA Experimental
+/-	python
	CCCL C Parallel Library
	Catch2Helper

🏃‍ Runner counts (total jobs: 1)

#	Runner
1	`linux-amd64-gpu-v100-latest-1`

1. Use pytest fixture to reuse computed inc_paths 2. Split long test into smaller individual tests with descriptive names 3. Add tests to check that directories contain expected marker files (versions) 4. Avoid using c4l in favor of cccl

github-actions · 2025-01-30T20:44:28Z

🟩 CI finished in 45m 37s: Pass: 100%/1 | Total: 45m 37s | Avg: 45m 37s | Max: 45m 37s

🟩 python: Pass: 100%/1 | Total: 45m 37s | Avg: 45m 37s | Max: 45m 37s

🟩 cpu
  🟩 amd64              Pass: 100%/1   | Total: 45m 37s | Avg: 45m 37s | Max: 45m 37s
🟩 ctk
  🟩 12.6               Pass: 100%/1   | Total: 45m 37s | Avg: 45m 37s | Max: 45m 37s
🟩 cudacxx
  🟩 nvcc12.6           Pass: 100%/1   | Total: 45m 37s | Avg: 45m 37s | Max: 45m 37s
🟩 cudacxx_family
  🟩 nvcc               Pass: 100%/1   | Total: 45m 37s | Avg: 45m 37s | Max: 45m 37s
🟩 cxx
  🟩 GCC13              Pass: 100%/1   | Total: 45m 37s | Avg: 45m 37s | Max: 45m 37s
🟩 cxx_family
  🟩 GCC                Pass: 100%/1   | Total: 45m 37s | Avg: 45m 37s | Max: 45m 37s
🟩 gpu
  🟩 v100               Pass: 100%/1   | Total: 45m 37s | Avg: 45m 37s | Max: 45m 37s
🟩 jobs
  🟩 Test               Pass: 100%/1   | Total: 45m 37s | Avg: 45m 37s | Max: 45m 37s

👃 Inspect Changes

Modifications in project?

	Project
	CCCL Infrastructure
	libcu++
	CUB
	Thrust
	CUDA Experimental
+/-	python
	CCCL C Parallel Library
	Catch2Helper

Modifications in project or dependencies?

	Project
	CCCL Infrastructure
	libcu++
	CUB
	Thrust
	CUDA Experimental
+/-	python
	CCCL C Parallel Library
	Catch2Helper

🏃‍ Runner counts (total jobs: 1)

#	Runner
1	`linux-amd64-gpu-v100-latest-1`

leofang · 2025-01-31T04:31:16Z

Q: How much efforts is needed to also build cuda.parallel with skbuild-core? I think it was the main mission of #2334 because it contains the C library to be built with nvcc.

oleksandr-pavlyk · 2025-01-31T13:35:39Z

Oh, you are right. This does not close gh-2334..

Transitioning cuda.paralallel to scikit-build-core requires decoupling the building of cccl-c library and making it an imported dependency. I should perhaps make this change in a separate PR, but the claim of closing gh-2334 needs to be removed

oleksandr-pavlyk · 2025-01-31T14:30:43Z

It appears there is no issue to change cuda_cccl build system, and each PR must close an issue. So should one be created?

oleksandr-pavlyk · 2025-01-31T14:43:00Z

Actually, I do not think this is ready yet @leofang

This PR adds an empty file cuda/__init__.py which clobbers the cuda/__init__.py from cuda-python package. This is not good. Also, multiple packages installing into the ${PREFIX}/lib/pythonX.Y/site-packages/cuda/ is not a good practice because of possibility of clobbering.

So cuda/__init__.py must be removed.

leofang · 2025-01-31T15:30:57Z

Good catch, yes I misread python/cuda_cccl/cuda/__init__.py and thought it's installed as cuda_cccl/cuda/__init__.py. This needs to be fixed. cuda is a namespace package (NVIDIA/cuda-python#75).

oleksandr-pavlyk · 2025-01-31T17:56:52Z

I pushed a change that removes cuda/__init__.py. I also changed build system for cuda_parallel to scikit-build-core as well. This time the change should indeed close gh-2334 :)

github-actions · 2025-01-31T18:53:17Z

🟩 CI finished in 46m 01s: Pass: 100%/1 | Total: 46m 01s | Avg: 46m 01s | Max: 46m 01s

🟩 python: Pass: 100%/1 | Total: 46m 01s | Avg: 46m 01s | Max: 46m 01s

🟩 cpu
  🟩 amd64              Pass: 100%/1   | Total: 46m 01s | Avg: 46m 01s | Max: 46m 01s
🟩 ctk
  🟩 12.6               Pass: 100%/1   | Total: 46m 01s | Avg: 46m 01s | Max: 46m 01s
🟩 cudacxx
  🟩 nvcc12.6           Pass: 100%/1   | Total: 46m 01s | Avg: 46m 01s | Max: 46m 01s
🟩 cudacxx_family
  🟩 nvcc               Pass: 100%/1   | Total: 46m 01s | Avg: 46m 01s | Max: 46m 01s
🟩 cxx
  🟩 GCC13              Pass: 100%/1   | Total: 46m 01s | Avg: 46m 01s | Max: 46m 01s
🟩 cxx_family
  🟩 GCC                Pass: 100%/1   | Total: 46m 01s | Avg: 46m 01s | Max: 46m 01s
🟩 gpu
  🟩 v100               Pass: 100%/1   | Total: 46m 01s | Avg: 46m 01s | Max: 46m 01s
🟩 jobs
  🟩 Test               Pass: 100%/1   | Total: 46m 01s | Avg: 46m 01s | Max: 46m 01s

👃 Inspect Changes

Modifications in project?

	Project
	CCCL Infrastructure
	libcu++
	CUB
	Thrust
	CUDA Experimental
+/-	python
	CCCL C Parallel Library
	Catch2Helper

Modifications in project or dependencies?

	Project
	CCCL Infrastructure
	libcu++
	CUB
	Thrust
	CUDA Experimental
+/-	python
	CCCL C Parallel Library
	Catch2Helper

🏃‍ Runner counts (total jobs: 1)

#	Runner
1	`linux-amd64-gpu-v100-latest-1`

python/cuda_parallel/pyproject.toml

shwina

LGTM (but would also wait for Bradley & Leo to approve before merging)!

Can we change the title of this PR to:

Transition build system of cuda_cccl and cuda_parallel to scikit-build-core

Awesome work @oleksandr-pavlyk!

oleksandr-pavlyk · 2025-01-31T20:19:44Z

@leofang Overall, do we have tests that various python packages installing into site-packages/cuda do not clobber one another? This could be tested by inspecting RECORD files from their wheels.

leofang · 2025-02-02T16:02:17Z

Overall, do we have tests that various python packages installing into site-packages/cuda do not clobber one another? This could be tested by inspecting RECORD files from their wheels.

I think the Kitmaker team will eventually handle this test. For now we are on our own.

leofang · 2025-02-02T16:04:48Z

@NVIDIA/cccl-cmake-codeowners we still need another review here. There's no change to other projects CMakeList; new additions here are to serve the new build system (skbuid-core).

rwgk · 2025-02-03T22:50:02Z

python/cuda_cccl/test/test_cuda_cccl.py

@@ -0,0 +1,63 @@
+import pytest


Nice addition!

Is this run in the CI?

(For cuda_parallel and cuda_cooperative the tests are run from ci/test_python.sh)

oleksandr-pavlyk added 4 commits January 29, 2025 17:46

Rewrite of cuda_cccl build system

f3fbf3a

Transition from using setuptools to using scikit-build-core, so that CMakeLists.txt is used to ensure that include is the same as in the cuda toolkit. Added test folder to test that the package works correctly.

Make project.version metadata dynamic

c000604

Use suggestion by Bradley to use scikit-build-core regex facility to extract version value from "_version.py"

Delete existing cuda_cccl

b2be856

Rename new_cuda_cccl to cuda_cccl to become ready to open a PR

dd191f7

oleksandr-pavlyk requested review from a team as code owners January 30, 2025 00:42

oleksandr-pavlyk requested review from miscco and gevtushenko January 30, 2025 00:42

shwina reviewed Jan 30, 2025

View reviewed changes

python/cuda_cccl/test/test_cuda_cccl.py Outdated Show resolved Hide resolved

Fix build break

a00a02f

Expand tests for cuda_cccl

792fd5d

1. Use pytest fixture to reuse computed inc_paths 2. Split long test into smaller individual tests with descriptive names 3. Add tests to check that directories contain expected marker files (versions) 4. Avoid using c4l in favor of cccl

leofang approved these changes Jan 31, 2025

View reviewed changes

leofang self-requested a review January 31, 2025 15:31

Change cuda_parallel over to use scikit-build-core

d392111

Delete unused _setup.py

8e76841

oleksandr-pavlyk requested review from bdice and shwina January 31, 2025 19:53

shwina reviewed Jan 31, 2025

View reviewed changes

python/cuda_parallel/pyproject.toml Show resolved Hide resolved

shwina approved these changes Jan 31, 2025

View reviewed changes

oleksandr-pavlyk changed the title ~~Transition build system of cuda_cccl to scikit-build-core~~ Transition build system of cuda_cccl and cuda_parallel to scikit-build-core Jan 31, 2025

leofang approved these changes Feb 2, 2025

View reviewed changes

alliepiper approved these changes Feb 3, 2025

View reviewed changes

shwina merged commit c52fcd0 into NVIDIA:main Feb 3, 2025
21 of 24 checks passed

oleksandr-pavlyk deleted the transition-cuda-cccl-to-scikit-build-core branch February 3, 2025 22:49

rwgk reviewed Feb 3, 2025

View reviewed changes

rwgk mentioned this pull request Feb 7, 2025

Remove cuda/__init__.py in cuda-parallel package #3750

Merged

2 tasks

oleksandr-pavlyk mentioned this pull request Feb 10, 2025

[BUG]: Editable builds of cuda_cccl and cuda_parallel broke after merge of pull/3597 #3761

Closed

1 task

Transition build system of cuda_cccl and cuda_parallel to scikit-build-core #3597

Transition build system of cuda_cccl and cuda_parallel to scikit-build-core #3597

Conversation

oleksandr-pavlyk commented Jan 30, 2025

Description

Checklist

github-actions bot commented Jan 30, 2025

🟥 python: Pass: 0%/1 | Total: 4m 36s | Avg: 4m 36s | Max: 4m 36s

👃 Inspect Changes

Modifications in project?

Modifications in project or dependencies?

🏃‍ Runner counts (total jobs: 1)

leofang commented Jan 30, 2025

github-actions bot commented Jan 30, 2025

🟥 python: Pass: 0%/1 | Total: 6m 03s | Avg: 6m 03s | Max: 6m 03s

👃 Inspect Changes

Modifications in project?

Modifications in project or dependencies?

🏃‍ Runner counts (total jobs: 1)

github-actions bot commented Jan 30, 2025

🟩 python: Pass: 100%/1 | Total: 47m 48s | Avg: 47m 48s | Max: 47m 48s

👃 Inspect Changes

Modifications in project?

Modifications in project or dependencies?

🏃‍ Runner counts (total jobs: 1)

github-actions bot commented Jan 30, 2025

🟩 python: Pass: 100%/1 | Total: 45m 37s | Avg: 45m 37s | Max: 45m 37s

👃 Inspect Changes

Modifications in project?

Modifications in project or dependencies?

🏃‍ Runner counts (total jobs: 1)

leofang commented Jan 31, 2025

oleksandr-pavlyk commented Jan 31, 2025

oleksandr-pavlyk commented Jan 31, 2025

oleksandr-pavlyk commented Jan 31, 2025

leofang commented Jan 31, 2025

oleksandr-pavlyk commented Jan 31, 2025

github-actions bot commented Jan 31, 2025

🟩 python: Pass: 100%/1 | Total: 46m 01s | Avg: 46m 01s | Max: 46m 01s

👃 Inspect Changes

Modifications in project?

Modifications in project or dependencies?

🏃‍ Runner counts (total jobs: 1)

shwina left a comment

Choose a reason for hiding this comment

oleksandr-pavlyk commented Jan 31, 2025

leofang commented Feb 2, 2025

leofang commented Feb 2, 2025

rwgk Feb 3, 2025

Choose a reason for hiding this comment